Multilingual Training and Cross-lingual Adaptation on CTC-based Acoustic Model

Phoneme-based multilingual training and different crosslingual adaptation techniques for Automatic Speech Recognition (ASR) are explored in Connectionist Temporal Classification (CTC)-based systems. The multilingual model is trained to model a universal IPA-based phone set using CTC loss function. While the same IPA symbol may not correspond to acoustic similarity, Learning Hidden Unit Contribution (LHUC) is investigated. Given the multilingual model, different approaches are exploited and compared to adapt the multilingual model to a target language with limited adaptation data. In addition, dropout during cross-lingual adaptation is also studied and tested in order to mitigate the overfitting problem. Experiments show that the performance of the universal phoneme-based CTC system can be improve by apply LHUC and it is extensible to new phonemes during cross-lingual adaptation. Updating all the parameters shows consistently improvement on limited data. Applying dropout during adaptation can further improve the system and achieve competitive performance with Deep Neural Network (DNN)/ Hidden Markov Model (HMM) systems even on 21 hours data.